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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 


A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH{S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
eamed patent temi adjustment. See 37 CFR 1.704(b). 

Status 

1 )S Responsive to communication(s) filed on 06 June 2005 . 
2a)n This action is FINAL.. 2b)S This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims ' 

4) ^ Claim(s) U26 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. . 

5) \3 Claim(s) is/are allowed. 

6) ^ Claimfs) 1-26 is/are rejected. 
?)□ Claim(s) is/are objected to. 

8) n Claim{s) are subject to restriction and/or election requirement. 

Application Papers 

9) n The specification is objected to by the Examiner. 

10)^ The drawing{s) filed on 22 July 2004 is/are: ajD accepted or b)KI objected to by the Examiner 
Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction Is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 !)□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priprity under 35 U.S.C. § 119 

12)13 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 
a)|g| All bjn Some * c)^ None of: 

1 ,0 Certified copies of the priority documents have been received. 

2. n Certified copies of the priority documents have been received in Application No. : . 

3. E1 Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 
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DETAILED ACTION 
Drawings 

1 . The drawings are objected to because ttie boxes of Figs. 1,14, and 16 
illustrating the system do not have descriptive labels. Corrected drawing sheets in 
compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid 
abandonment of the application. Any amended replacement drawing sheet should 
include all of the figures appearing on the immediate prior version of the sheet, even if 
only one figure is being amended. The figure or figure number of an amended drawing 
should not be labeled as "amended." If a drawing figure is to be canceled, the 
appropriate figure must be removed from the replacement sheet, and- where necessary, 
the remaining figures must be renumbered and appropriate changes made to the brief 
description of the several views of the drawings for consistency. Additional replacement 
sheets may be necessary to show the renumbering of the remaining figures. Each 
drawing sheet submitted after the filing date of an application must be labeled in the top 
margin as either "Replacement Sheet" or "New Sheet" pursuant to 37 CFR 1.121(d). If 
the changes are not accepted by the examiner, the applicant will be notified and 
informed of any required corrective action in the next Office action. The objection to the 
drawings will not be held in abeyance. 
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Claim Rejections • 35 USC § 102 

The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 

Claim 1 - 8, 11 - 15, 18 - 22. 25 are rejected under 35 U.S.C. 102(b) as being 
anticipated by "Evaluation of Artificial Reality: Chapter 1". 

2. In regards to claims 1 -4, 11, 12, 18, 19, 25 - 

The system is for superimposing an outer appearance formed by CG on an 
assistive robot performing tasks indoor. The error of the CG display position is such 
that the size of the displayed CG is increased larger than the actual robot [pg. 18, lines 
7-15]. This increased size of the display CG corresponds to the nimbus as disclosed 
(claims 1 - 4, 11, 12, 18, 19, 25: a nimbus generating device for generating a 
nimbus image around a periphery of ttie computer graphics). The configuration of 
an augmented reality image display system is shown in Fig. 1.32. Based on the visual 
point position and attitude information obtained from a three-dimensional position 
sensor attached to the head, CG are generated by a computer (claims 3, 4, 12, 19, 25: 
an image generating device for generating computer graphics in a figure 
corresponding to a figure of an actual object and a positional relationship 
between an observer, who is viewing the actual object, and the actual object) [pg. 
10, lines 24 - 25; pg. 11, line 1]. Furthermore, the CG can be selected by taking the 
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form of a character that the user likes, a deceased spouse, or the Wke (claims 1,2,11, 
18: an image selecting device for selecting computer graphics in a figure of an 
actual object and a positional relationship between an observer, who is viewing 
the actual object, and the actual object) [pg. 7, lines 1 - 3; Fig. 1.3]. Simultaneously, 
a real image of the background is obtained from a CCD camera, and using the 
luminance key synthesis of a video mixer (claims 1 - 4, 11, 12, 18, 19: a combined 
image generating device for generating a combined image combining the 
graphics and the nimbus image), each are synthesized and displayed on an HMD as 
augmented reality (claims 1-4, 25: an image display processing device for 
displaying the combined image on an display which is viewed by the observer so 
that the combined image is superimposed on the actual object) [pg. 1 0, lines 24 - 
25; pg. 1 1 , lines 1 - 4]. The video mixer also doubles as the combined image sending 
device as disclosed in claims 2 and 4, where as shown in Fig. 1.32, the video mixer 
sends the combined image of the CG and real image obtained by the camera to the 
HMD (claims 2, 4: a combined image sending device for sending the combined 
image to the observer side). Furthermore, in this system, an lsotrak2 (made by 
Polhemus) is used as the magnetic three-dimensional position sensor. The computer is 
a TD-30 (made by Intergraph; CPU: Pentium 133 MHzx2, with built-in Open GL 
graphics accelerator), and an FS5 (made by Virtual Research Systems) is used as the 
HMD [pg. 11, lines 5-10]. These provide the means as disclosed in claims 18 and 19. 
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3. In regards to claims 5, 13, 20 - 

The system is for superimposing an outer appearance formed by CG on an 
assistive robot performing tasks indoor. Since the total width of the robot perfomiing a 
carrying task or the like is about 500 mm, and moreover, the robot is used indoors, it is 
assumed that it operates nearly 2 m in front of the user. The error of the CG display 
position is such that the size of the displayed CG is increased larger than the actual 
robot by as much as this error [pg. 18, lines 7 - 15]. This increased size of the display 
CG corresponds to the nimbus as disclosed. The maximum position error corresponds 
to the positional error when the robot is some distance away. The CG can be increased 
to counterbalance this error (claims S, 13, 20: wherein said nimbus generating 
device estimates an error in measurement of a distance between the observer 
and the actual object, and generates based on the error the nimbus image having 
an adequate thicl<ness for preventing the actual object from being seen 
protruding from a periphery of the computer graphics) [pg. 18, lines 19 - 22]. 

4. In regards to claims 6, 14, 21 - 

From the example cited in claim rejections 5/1 , 5/3, the maximum error (22 mm) 
corresponds to the positional error (44 mm) when the robot is 2 m away from the user. 
The resulting error is about one-tenth the size of the robot, and the CG can be 
increased to counterbalance this error. By correcting the three-dimensional position 
sensor, it is thought that the error can be controlled within a substantially permissible 
range (claims 6, 14, 21: wherein said nimbus generating device generates the 
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nimbus image having an adequate thickness for not showing the observers 
displacement which occurs between the actual and the computer graphics when 
the actual object or the observer moves) [pg. 18, lines 16 - 20; pg. 19, line 1]. 

5. In regards to claims 7, 15, 22 - 

In order to superimpose the CG on the robot and synthesize an image without a 
feeling of incompatibility, accuracy was required of the three-dimensional position 
sensor [pg. 12, lines 1 - 3]. As seen in Fig. 1.34, the rotation of the head to the right 
and left is measure by the sensor as a change in angle of direction, the tilting back and 
forth is measured as a change in the angle or elevation, and the tilting to right and left is 
measured as a change in the angle of torsion (claims 7, 15, 22: a detecting device for 
detecting at least one of distance and an angle between a display with is viewed 
by the observer and the actual object and a direction of the actual object seen 
from the observei) [pg. 13, lines 4 - 8]. 

6. In regards to claim 8 - 

The CG of a human shape on an arm-type care-giving robot used for a carrying 
task actually located at a position 2m away from the user [pg. 20, lines 7 - 9]. 
Furthermore, although a CG was superimposed on a target robot standing still, the 
robot actually changes position and attitude over time (claim 8: wherein the actual 
object is a humanoid robot or an animal-type robot other than a human, which 
can move freely) [pg. 22, lines 7- 9]. 
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Claim Rejections • 35 USC § 103 

The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

7. Claims 9, 16, 23 rejected under 35 U.S.C. 103(a) as being unpatentable over 

"Evaluation of Artificial Reality: Chapter 1" in view of Yuasa et al. (US Patent No. 

6,184.888 61). 

"Evaluation of Artificial Reality: Chapter 1" teaches the limitations of claims 9, 16, 
and 23 with the exception of disclosing receiving the computer graphics from outside via 
a network. However, Yuasa et al. teaches an image rendering apparatus and method 
intended for use in an environment in which the data transfer rate is limited, as in a 
network such as the Internet. 

"Evaluation of Artificial Reality: Chapter 1" discloses a system for superimposing an 
outer appearance formed by CG on an assistive robot performing tasks indoor. The 
error of the CG display position is such that the size of the displayed CG is increased 
larger than the actual robot [pg. 18, lines 7-15]. This increased size of the display CG 
corresponds to the nimbus as disclosed. The configuration of an augmented reality 
image display system is shown in Fig. 1 .32. Based on the visual point position and 
attitude information obtained from a three-dimensional position sensor attached to the 
head, CG are generated by a computer [pg. 10, lines 24 - 25; pg. 1 1, line 1]. 
Furthermore, the CG can be selected by taking the fomri of a character that the user 
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likes, a deceased spouse, or the like [pg. 7, lines 1 - 3; Fig. 1.3]. Simultaneously, a real 
image of the background is obtained from a CCD camera, and using the luminance key 
synthesis of a video mixer, each are synthesized and displayed on an HMD as 
augmented reality [pg. 1 0, lines 24 - 25; pg. 1 1 , lines 1 - 4]. The video mixer also 
doubles as the combined image sending device, where as shown In Fig. 1.32, the video 
mixer sends the combined image of the CG and real image obtained by the camera to 
the HMD. Furthennore, in this system, an lsotrak2 (made by Polhemus) is used as the 
magnetic three-dimensional position sensor. The computer is a TD-30 (made by 
Intergraph; CPU: Pentium 133 MHzx2, with built-in Open GL graphics accelerator), and 
an FS5 (made by Virtual Research Systems) is used as the HMD [pg. 11, lines 5-101. 

Yuasa et al. teaches an image rendering apparatus and method intended for use in 
an environment in which the data transfer rate is limited, as in a network such as the 
Internet. As shown in Fig. 1, the rendering apparatus (100) resides in the computer 
(102). The rendering apparatus may form part of a graphics card located in the 
computer or may be a hardware accessory that plugs into or is othenwise connected to 
the graphics card of the computer [col. 6, lines 28 - 481. The computer is shown 
connected via the I/O port (104) as a client of the network (106). The I/O port is 
connected directly or indirectly to the bus (108) to which other elements of the rendering 
apparatus and the computer are connected. The network, I/O port and bus constitute a 
path through which the rendering apparatus receives three-dimensional graphics data 
including texture data defining one or more textures (claims 9, 16, 23: comprising a 
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communication device for receiving Uie computer graphics from outside via a 

network) [col. 6, lines 39 - 46]. 

Therefore, it. would have been obvious to one of ordinary skill in the art at the time 
the invention was made to include the rendering apparatus of Yuasa et al. as a part of 
the Open GL graphics accelerator built-in the computer disclosed in "Evaluation of 
Artificial Reality: Chapter 1", as well as incorporate the I/O port of Yuasa et al. to 
connect the computer disclosed in "Evaluation of Artificial Reality: Chapter 1" to a 
network because this would allow the user to import images from other sources, such 
as the internet, when the user is opting to select the CG by taking the form of a 
character that the user likes, a deceased spouse, or the like ["Evaluation of Artificial 
Reality: Chapter 1": pg. 7, lines 1 - 3; Fig. 1.3]. Furthemnore, importing the images from 
another source provides a larger range of characters the user may choose from as well 
as saving memory space on the computer by not having to generate and save 
multitudes of characters. 

8; Claims 10, 17, 24, 26 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over "Evaluation of Artificial Reality: Chapter 1 ". 

"Evaluation of Artificial Reality: Chapter 1" teaches the limitations of claims 10, 17, 
24, 26 except disclosing a second actual object. However, "Evaluation of Artificial 
Reality: Chapter 1" teaches the CG of a human shape is superimposed on an ann-type 
care-giving robot used for a carrying task [pg. 20, lines 7 - 9]. The claim rejections for 
claims 1 - 4, 1 1 , 12, 1 8, 1 9, and 25 are incorporated as reference herein. The camera 
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as shown in Fig. 1.32 will record the robot carrying the item related to the carrying task 
and the video mixer superimposes the CG of a human shape on top of the robot. Since 
the CG is only of the human shape and does not contain a CG of the item related to the 
carrying task, it would have been obvious to one of ordinary skill in the art that the work 
station shown in Fig. 1 .32 provides a mask of some type to reveal the item related to the 
carrying task so that the super-imposed image sent to the HMD via the video mixer 
displays the item to the user so the robot looks like a character the user finds comforting 
performing the task (claims 10, 17, 24, 26: wherein the computer graphics have a 
lacking area for showing the observer a second actuai object having a part which 
exists on the observer side than the actual objecf). Although a CG was 
superimposed on a target robot standing still, the robot actually changes position and 
attitude over time [pg. 22, lines 7- 9]. The error of the CG display position is such that 
the size of the displayed CG is increased larger than the actual robot [pg. 18, lines 7 - 
1 5]. This error, i.e. nimbus, compensates for the possible movement of the robot so 
when the robot moves, there is no chance that the actual robot will be seen apart from 
the CG superimposed. Thus, if the robot were carrying an item, it would have been 
obvious to include a nimbus around the item for the same reason (claims 10, 17, 24, 26: 
wherein said nimbus generating device also generates a nimbus image around a 
periphery of the lacking area). 
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Conclusion 


Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Michelle K. Lay whose telephone number is (571) 272- 


supervisor, Michael Razavi can be reached on (571) 272-7664. The fax phone number 
for the organization where this application or proceeding is assigned Is (571) 273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). 

Michelle K. Lay 
Patent Examiner 
Art Unit 2672 


7661 . The examiner can normally be reached on Monday - Friday, 7:00am - 4:30pm. 


If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
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